Fixes #605 - Run PostgreSQL tools in containers - #906

reviews:
  auto_review:
    drafts: true

No actionable comments were generated in the recent review.

Merge Risk: Minimal · previous review cc487

The containerized PostgreSQL commands preserve the documented connection and backup paths, including restore password forwarding. No actionable merge-blocking risk was identified; reported integration results were not independently reproduced.

Security architecture risk: Moderate · up to cc487

Backup and restore now entrust database credentials and backup contents to a container image selected by a mutable tag. Temporary containers and read-only restore mounts limit exposure, but a compromised image could access sensitive data or alter database operations. The image was already trusted for internal PostgreSQL deployment; this change extends its authority to backup archives and external-database backups.

Retained concerns

• Medium · security · inferred: A mutable PostgreSQL image becomes a credential-bearing backup client with host-network access and mounted backup storage, including for external-database backups. A compromised image could disclose database credentials, tamper with writable dumps, or read restore archives. The image already had internal-server authority before this PR, but these client and archive exposures are new. No digest constraint is visible in the inspected configuration; deployment-enforced image verification remains unknown.

Security review details

Security Blast Radius

• inferred — A compromised client image could affect the configured databases within the forwarded accounts' privileges, the mounted backup directory, and services reachable through the host network. Restore supplies database-administrator authority. The inspected commands do not establish unrestricted host-filesystem access or cross-environment compromise.

Security Findings and Attack Paths

• inferred — The supported attack path requires control of a selected image release or its delivery: image execution then exposes forwarded credentials and mounted data. No ordinary unprivileged request path or image compromise was demonstrated. Existing internal-server trust limits the incremental concern there, but does not cover newly mounted archives or external-database backup credentials.

Trust Boundaries and Controls

• observed — Argument arrays avoid shell interpretation. Containers request neither --privileged nor added capabilities, use temporary-container removal, and restrict restore-volume writes. These controls do not prevent the selected image from reading its environment, accessing mounted contents, or using the host network.

Resilience and Maintainability Implications

• inferred — Pre-role image preparation limits failures from initial image acquisition before backup service shutdown or database replacement. It does not make restore atomic: the existing drop, create, and restore sequence can leave incomplete databases after failure. No PR-specific atomicity regression was established.

Hardening Proposals

• proposed — Bind credential-bearing client execution to an approved immutable image identity and verified provenance. Record that identity for reproducible recovery, and validate effective image-user, capability, and concurrent SELinux-mount behavior before treating these containers as a stronger isolation boundary.

Pre-merge checks: 4

Check name	Status	Explanation
Linked Issues check		The changes satisfy issue [#605]. pg_dump, pg_isready, and pg_restore now run through podman run with the configured PostgreSQL image. The tasks use host networking and mount backup paths only…
Out of Scope Changes check		The changed playbooks, backup and restore tasks, and integration test directly support issue [#605]. Image availability is required for container execution. The host-tool checks verify the removal of …
Description check		The description clearly explains the move of PostgreSQL tools into containers, removal of host package dependencies, testing, and related compatibility changes.
Title check		The title clearly and concisely identifies the main change: running PostgreSQL tools in containers.

Full details: Linked Issues check

Explanation

The changes satisfy issue [#605]. pg_dump, pg_isready, and pg_restore now run through podman run with the configured PostgreSQL image. The tasks use host networking and mount backup paths only for dump and restore operations. The backup and restore playbooks ensure that the client image is available and remove the host postgresql package installation. The integration test verifies that pg_dump and pg_isready are not installed on the host.

Full details: Out of Scope Changes check

Explanation

The changed playbooks, backup and restore tasks, and integration test directly support issue [#605]. Image availability is required for container execution. The host-tool checks verify the removal of the host dependency. No unrelated change is demonstrated by the pull request diff.

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

I investigated the two current red jobs; neither failure is caused by this PR's PostgreSQL tooling change:

• proxy deployment on CentOS Stream 9 with fapolicyd failed while downloading containers.podman from Ansible Galaxy with IncompleteRead(67561 bytes read, 17577 more expected). The deployment never started.
• upgrade from 2.y-stable reached Pulp migration, then the published quay.io/foreman/pulp:foreman-3.19 image failed to import Psycopg because libpq is missing. The sosreport confirms image digest a8677e83... and pulpcore-manager-migrate.service fails before connecting to PostgreSQL.

I reproduced the second failure directly against the currently published image and reopened pulp-oci-images #115 until the stable tag is rebuilt with the already released packaging fix.

No code change to this PR is indicated by either failure. Once the full workflow finishes and the stable Pulp tag is repaired, I will trigger a fresh matrix run. The PR's own lint and the live backup/restore validation remain green.

The remaining red matrix is now covered by the shared follow-ups rather than this PostgreSQL-container change: #910 handles recovered Dynflow workers, #893 covers the CentOS Stream 9 fapolicyd cache, #863 fixes the FIPS/backup privilege-escalation timeout, and pulp-oci-images#115 tracks the broken 3.19 Pulp image used by the upgrade job. This PR’s lint, Packit, external-database, CentOS Stream 10 security, capsule and targeted live backup/restore validation are green.

Note that #605 talks about an example, but that we should consider how to address the use of PostgresSQL tools inside the container as a whole. This only touches a subset of that and does it in a very brute force method and does not consider why we currently use the postgresql collection, the benefits there in, or how to avoid re-implementing the same large podman run commands each and every time there is some interaction with PostgreSQL in the code base.

For me, this is an example of an issue that needs design discussion, not just trying to throw an agentic solution at it.

I focused too narrowly on dump/restore. The design needs to cover database/user/extension management and readiness checks as well, and retain the community.postgresql modules rather than replace their behavior with repeated CLI commands. I would start by agreeing on a shared container execution mechanism for those modules and the few CLI-only operations, including internal/external databases, credentials and air-gapped image selection. This PR is not that complete solution yet, so I have left the implementation unchanged rather than treating another CI rerun as an answer to the design concern.

Fixed the backup test setup in 4eb6b8b. The IOP-enabled run had 466 passing tests but 14 backup setup errors because pg_dump was installed on the host. That is a valid deployment, so the fixture no longer rejects it. It still runs the backup and verifies the resulting dumps and metadata.

Also replaced the removed postgresql_ext parameter alias db with login_db, supported by the minimum collection version 4.0. Targeted Ansible lint passed across 77 files with no rule findings; Ruff and git diff --check passed. Rebased on current master; full deployment and backup CI is running again.

CI follow-up (28 October): the completed run had two failing jobs:

• Upgrade: pulling the IOP remediation image from Quay ended with unexpected EOF; the dependent service could not start. This happened before backup validation.
• EL10 fapolicyd: Nmap could not open its outlib.lua library. The audit log confirms a fapolicyd denial (subj_trust=0, obj_trust=0) for Nmap. This is a test-tool access failure, not a failed TLS negotiation. That job passed 329 tests, including its applicable backup checks.

The other 22 jobs passed. I refreshed the unchanged code as 7018c98 for one further CI attempt, preserving the tree, parent and single-commit PR. The new result is pending.